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'.Evaluation of the Field Jest of Project Information Packages (PIPs) 

4 * ; , - — - — — 

In its continuing search for successful means by which to disseminate ' 
exemplary education projects, the U.S. Office of Education 1n 1973 
supported the development of six Project Information Packages (PIPs). 
The PIPs were designed to provide "how to" information and Instruc- 
tions to facilitate the implementation of the selected compensatory 
reading and mathematics projects in new school districts with a 
minimum amount of technical assistance. The projects selected for 
packaging parsed stringent criteria of effectiveness with respect to 
reading and^mathematics sk1l]s in addition to review by the Education 
division's Joint Dissemination Review Panel*. Five of the six projects, 
originally developed by local education agencies with ESEA, Title I 
funds, are "pull out" projects. These projects serve a special target 
group of students and therefore supplement rathef than replace the 
regular classroom teaching. Consequently, the projects require students 
to leave class and go to another location to participate. The sixth 
project* originally developed by a local education agency with other 
federal funds, is not a "pull out" type of project but instead serves 
1 all chfldren in the specif i§<L grade and therefore requires regular 
classroom teachers to make changes in their methods or behavior. 

The central principle assumed in developing the PIPs*was that if . . 
the antecedent conditions of the effective instructional project 
could be established in a new site (the Replication Principle), then 
the project would be reproduced and woulcl again prove effective 1n 
terms of student achievement gains. In addition to thi!s principle 
two other assumptions were made. First* it was assumed that 1t 
was necessary to match the setting of the replicating site with the setting 
of the original » successful site. This was to be accomplished by provld* 
ing information to potential project adopter sites about the original 
settings in the fora of an Analysis. and Selection Kit (ASK). Districts 
interested in replicating a packaged project would use the ASK and 
self select an appropriate project that matched local conditions, 
i Second » it was assumed that project management was the key to 
replicating the original conditions. Given this assumption^ the PIPs 
highlighted the importance of a dynamic, experienced project director — 
who has responsibility for establishing the antecedent conditions for 
effective instruction. Information provided In the PIPs toas manage- 
ment oriented to heTp project directors and teachers set up the 



conditions for th£ instructional program to J>e recreated; the 
packages specj/ied requirements for space, qualified staff, i 
-jnaier.ials.juw .equipment, $t udent£-4^ect4flry^(^ — 
keeping aryf the Tike. The PIPs did not describe in detail the 

teaching/iearning episode, classroom. interactions* or 

sequenced of events within the instructional program nor did thev 
describe the appropriate uses of each of the recommended curriculum 
mater/als^since it was assumed that these events would follow 
give*/ the appropriate mix of resources e^nd the required teaching 
staff of highly qualified, well trained compensatory reading or 
mathematics specialists. 

In 7 fiscal year 1975, PIPs were tried out in a number of sites 
across the country. Under section 306 of E$£A> Title III, 
/tjrants were awarded to 19 school ^sites for the purpose of implementing 
one of the exemplary projects Via * PIP/ To assess the viability of 
disseminating successful projects via packages as a way of improving 
reading and mathematics skills of disadvantaged children, a contract 
was awarded to StanfoHl Research Institute in June The two year 

study was to "examine the implementation of the packaged projects 1n - 
the try out sites and focus oil /the following*ques-t1ons: 

■ * * * *. 

1. Are local education agencies motivated to adopt ^ 
a. packaged project? *,* 

2. Can exemplary projects be implemented in new sites ■ 
via the PIPs? Where Implementation problems are 

due to faults in the packages, can reasonable modifica- 
tions be recommended? * 

3". For what functions and in what amount is technical* 
assistance required? If a considerable^ amoijnt of 
technical assistance is required, can the packages 
be made more autonomous? * 

4. Are the projects, implemented'yia the PIPs, effective 
in improving student achievement? , * - * . 

5. What are the effects of the projects on student attitudes? 
^ Are the projects acceptable to the local education agency* - 

to teachers, parents, and the conmunity? 

* 

6. What is,,the coit of implementing the projects? 
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The first year of the study was to examine the Implementation process ^ 
and determinethe fidelity of implements tton _of the PlP^speclfications-, ----- * 
in-eactr-ofHhe 19 sites » Tdentity implementation problems encountered; 
by the tryout sites arid recommend how the packages might be revised 

* in lj.ght Jit the. problems Identified* Also of concern during the /* 
initial year of the study was-Uje determination of the usefulness of the 
PIPs for guiding project Implementation, the soundness of the principles f 
and assqmptloos upon which the PIPs were developed, and the reaction 
of project participants and non-participants to the^projects. The 
first year of this study has been completed and the findings are 
summarized here. The second year of the study vml investigate the 
impact of the projects on student achievement and will'explore the 

* participating school districts' intentions for continuing th^projects 
in school- year 1976-77 when Title III section 306 money 1s n^longer 
available for their operation- 

The Study Approach f 

The major evaluation strategy was to compane what was specified 1n V. 
4a t the PIPs, given the assumptions upon which they were developed, with 

/ " , what" actually occured at, the project sites- In order to investigate- 

/ the discrepancies betweert what was expected with actual- events, and *" 

/ *, the sources of those dfscrepancies several ,data collection activities 

. : were undertaken^ 

) t 

Five visits were made to each of the 19 field test.sftes throughout 
the year to obseiPVe the project, to interview project and non-project 
personnel involved in implementation, and to conduct student testfhg- 
Observations of the projects during the site visits were used to *. 

* determine the degree of implementation of the specified instructional 
program elements and environment- Interviews were conducted with 
district administrators, principals, teachers and aides to learn 
about the implementation process* ^determine the causes of imple- 
mentation problems, project deviations and modifications and to deter- 
mine acceptability of the packaged projects and the like. Informal 

, contacts with parents of cfmdre^who were participating were also 
conducted to determine their reaction to the projects- 

Although the first year of the evaluation was to focus, on project 
implementation- rather than Impact on participating students, student, 
pre- and post-measurements, both standardized achievement tests and 
attitude surveys were administered to a minimal sample of participat- 
ing 'Students, This was done as a check ta make sure th&t the imple- 
mentation yearywai not disruptive irkterms of student achievement and 
* attitudes . ^ 
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Contact report forms were used^throughout the year project staff 

at the original and try out project sites an&^y^^ — 

evaluation staff ty reportr^t^phwe~cbnversatTons , visits and other 
contacts where assistance or clarification was requested^ offered 
or obtained. Finally, Instructional Staff Questionnaires and Adminis- 
trative Staff Questionnaires wet:e administered to assess staff 
atti.tuGes ^toward the PIP and the local project and Resource/Cost t 
Questionnaires were administered to determine the resources and 
associated cost of project implementations. * 

Findings and Conclusions 

The resuT^of the first year of the study were described in terms of the 
stages involved. in the implementation process and the soundness of the 
assumptions and principles upon which the PIPs were developed., 

^. 

During, the selection/adoption stage, it was intended that local decision 
makers wduld: (1) utilize attareness materials— the Analysis and. 
Selection Kit- or the AS£--to determine if one of the packaged projects 
met local needs, (2) select an appropriate project to implement and *(3) 
apply for the ESEA, Title III, section 306 funds that were made available 
for PIP implementation. This stage did not proceed as planned, however, 
for a' number of reasons. Most important was the fact that awareness 
materials.-were not provided to potential adopters and what happened was 
that most local education agencies (LEAs) applied "blind' 1 fora grant 
and a PIP project they knew little or nothing about and agreed to 
"replicate" projects and cooperate with an evaluator. Packaged projects 
were assigned to LEAs applying for grants on the basis of information 
provided in 'their applications. As a result, there appeared to be 
much misinformation held about the packages and the exemplary projects * 
by the LEA staffs. This was due in part to the selection/adoption, pro- . 
cess that actually occurred and in part to the lack of cunrnunication 
^between the dissemination agent (USOE) and the participants. In 
addition, LEAs -were not clear about whom to contact when questions arose. 

In applying for a PIP, LEA personnel were notmotlvated by the concept 
that implementation of a proVea project as described "in the PIP would 
lead to similar success. Instead the motivating factor was a need to 
serve certain grade levels or special groups in the district with 
good educational programs and seeing th^ Title III gr^nt as a way to" 
serve those needs. However, because "replication was a condition of 
the grant, th£ greit majority of district staffs were willing to 
adhere to the instruction in the packages for Implementing the projects 
and were willing to adapt themselves in terms of staffing, scheduling, 
providing facilities, and the like as specified in the PIPs* 
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The actual selection/adoption processes that occurred made it dear 
that the role of a'dlssemination agent and the Lines of conmunica- 
tion among l particij>^ti__i_o_th^adoptiori process must be constdered t 
al^ng wTth the ASK and PIP material s* a$ necessary parts of a workable 
dissemination strategy. 

Ouring the start-up stage, as specified in the P I Ps t the'project 
staff is hired* appropriate space* equipment and materials are 
obtainedt the non-project staff are oriented to the «. 
project, and students are selected and scheduled for participation. 
For the staff at the try out sites these activities w^re difficult 
to conduct according to the sequences t time lines and procedural 
recommendations in the PIPs due to the late and piecemeal arrival 
of tHS PIP material s. It was because of the dedicated efforts of 
the local PIP project directors and staff that the start-up 
activities, which required several months* were condensed into 
several weeks* problems were resolved* and the projects became a 
reality. Thus* by the beginning of the school year or a few weeks 
thereafter* the main elements of the projects were present. 

It was in the areas of preservice training and orientation of non- 
project staff that the most critical EindVidespread deviations from' 
PIP specifications occurred. This was Partly due to the fact that the 
packages did not col^unicate enough about the nature of the Instruc- 
tional program to permit project directors to lead training sessions 
or orientations wfth confidences Consequently, a major recommendation 
for PIP revision tis the addition of more explicit information on 
training and instruction. \ 

An important finding related to the start-up, stage * which resulted 
in a major recofrmendation for revision of the PIPs* i^s ,that some 
adaptation of original project activities would always Jbe necessary 
because no two school sites are exactly alike and neither packages 
nor any other"mechanism can anticipate which aspects cannot be repli- 
cated at new , sites. For packaging t this means* that descriptions 
of desired outcomes of each stage and the rationale for- the outcomes 
should be described clearly but appropriate steps and activities to 
attain these outcomes be included as suggestions rather than specifi- 
cations,. New districts could then decide the cortsequences of any £ 
adaptations pr\ changes required. The recommendations for PIP revisf&ns 
are- based on/this notion. * / 

In every site t with one possible exception local staffs using 
just the descriptions and instructions of the PIP created the ^ 
projects that were explicitly described in the PIPs t in spite 
pf the difficulties encountered by the LEA personnel in attempt^ 
ing to implement the PIP specifications for the startup' stage. 
Moreover t across sites t classrooms with the same type of PIP project 
were very much alike,. Thus t/ under the conditions of the field test t 
every project was implemented in the new. sites and t with one or two 
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the package should appeal to the peopSle who will be Involved 1n 

imp] eme nting it apd the management strategy s h ould suit the * 

adopter LEA and the potential project director, 

— Xhe^niana^ftmeni^ orientation of the PIPs wasjfound to be their best 
and most successful feature. While the-exJsting PIPs need much 
revision, the principle of a management package for project installa- 
tion is sound. In the field-test si^es, the PIPs enabled the project * 
directors to get the projects into operation in a -few weeks. While 
the management approach was found to be most efficient for project 
installation, a strict management orientation at the instructional 
program level did not provide enough information about the teaching 
processes or the details of the instructional program. Recommenda- 
tions for revisions'were made accordingly. Although more detailed 
information in this area is recommended for PIP. revision* it is clear 
that good project directors and teachers must be selected for the 
project; use of the PIP along cannot make a mediocre staff into a 
good ones * 
* . 

Regarding the question of how much autonomy, i.e. the degree to which 
the personalized "how to" assistance is not required, packages can 
achieve depends on the intended strategy for disseminating the 
packages arid the degree to which adaptations of Jthe projects occur, 
A very small amount of "how to 11 assistance was required in the 
fipld test projects. There*were few contacts with the original 
sites for purposes of understanding how to implement the proce- 
dures that were specified. These contacts were made 
primarily by telephone when a specific set of questions arose 
and served the purposes of building confidence and assuring new staff 
that correct procedures were being employed. This is not tq say that 
in any future use of PIPs > district staf^ will not seek or use 
assistance. Cne must consider that the current implementation was done 
under a grant agreemept with the government. Therefore* implementing 
the particular project as specified in the PIP— without outside 
assistance if possible— was a condition the district staff accepted under 
the grant agreement. However* under another set of circumstances the \ * 
staff in implementing school districts would^ likely, contact a previous! 
user of the project and visit to see the project in operation. 

Reliance on the Replication Principle in developing the PIPs had con- 
sequences for the'way in which the field test was administered. Pro- 
jects were monitored for their adherence to the PIP specifications' and 
constrained against "deviations" and "modifications". While 1t Is 
clear 'that fidelity to PIP specifications can be very good (is it was 
in several try out sites) the rfesults have shown that exact replication 
or perfect fidelity is not possible. Some at least minor adaptations 
are necessary since rib- two, sites or individuals are exacts alike.. 
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In summary the study found that: 



JU__ Using just the descrip tions atid instructinps^w^ided-4iv4h^ 



packaged projects local project staffs implemented the projects 7 
With considerable fidelity to the SMClfl cations . Jto jthe . packages^ 



2. Local project staffs were 6nthus1astically*involved In the 
Implementation of the projects and the projects were a source of 
considerable pride, f , 

- 3. The parents and non-project school personnel In the try out- 
sites were supportive of the projects, % m ^ 

4,. Few features of the original project site se&ned Important ^ 
to match in sites Implementing the projects. Those that were 
Important are: x 

!• .establishment of the local project directors 
autonomy and authority, 

2, commitment to the project on the part of the people- 
---- involved in implementing <it, ; ■ . 

■ 3, statisfactlon with the management approach of the *: 
.project by the adopter*distrfct, 

5. The PIPs were relatively autonomous in terms of the 
amount of additional "how to" assistance required by .the 
try out sites, but staff at the try out sites desired 
personal contact with other persons who had used the 
0 projects. 

* ■ 

s 6. While most try out sites implemented the projects 
with considerable fidelity to PIP specif icatlons > some 
minor adaptations were and are likely always to be 
necessary since no^two, sites are exactly alike. 

7, While'genfirally try 6ut sites reported that their 
projects were successful and "Indicated Improvement on the 
< part of students in academic skills* self confidence* 
* motivation to learn;. attendance and behavior I and students * 
reported that they enjoyed participating* test scores for" 1 
this first year were not ."educationally sl^jnYf leant* 
according to the criterion employed, 



